* code: Versioning for hotmaps dataset

* code: Fix CLI not working for foreign owned records

* code: versioning for co2stop dataset

* code: versioning for nitrogen statistics datset

* code: fix wrong file name for REUSE.toml

* code: versioning for NUTS 2013 and NUTS 2021

* code: Add limited debug mode for CLI script

* code: Fix issues with datasets not being detected

* code: Fix minor bugs on CLI and outputs

* code: Versioning for eurostat energy balances

* code: Remove obsolete retrieve script, now part of retrieve.smk

* code: Versioning for eurostat household energy balances

* code: Fix missing replacements for eurostat energy balances

* code: Remove prompt for API key

* code: Fix typo for enspreso biomass dataset

* code: Fix issue with OSM dataset versioning

* code: Versioning for WDPA and WDPA marine

* code: Implement versioning for TYNDP base network

* code: Fix bug missing osm_version in base_network.py

* code: Properly fix the bug this time

* code: Implement versioning for powerplants / powerplantsmatching

* code/docs: Implement versioning for costs

* code: Implement versioning for cutouts, move to data/ , deprecate shared_cutouts

* Fix: Wrong URL for powerplants

* code: Implement versioning for HDD and runoff per country (retrieval only)

* code: Add build options for HDD and runoff per country (untested)

* code: Add option for monthly retrieval of cutouts while building

* code: Restore build_natura_raster.py script

* code: Implement versioning for natura raster + reintegrate build natura raster script

* data: Change URB POP dataset to Zenodo link

* code: Adjust link for GEM GSPT

* code: Versioning for GEM Europe Gas Tracker

* doc: Correct license attributions in REUSE.toml for data/

* code: Implement versioning for BFS vehicle data

* code: Implement versioning for BFS GDP and POP data

* doc: Update data_inventory.csv for all the datasets added

* code: Fix versions for HDD and runoff per country

* code: Fix rules for creating HDD and runoff per country

* code: Add special cutout config for HDD/runoff

* code: Fix monthly_requests for cutouts

* code: Add logic for building and versioning of mobility profiles from BASt

* data: Update data for mobility profiles from BASt

* code: Windows compatability for cutouts

* code: Deprecate automatic cutout bounds

* pre-commit formatting

* code: Remove cutout dir configuration option

* code: Fix option for archived mobility_profiles

* code: Cleanup unused stub

* config: Remove deprecated config option from configfiles

* code: Remove prompt for API key

* LUISA land cover database - data versioning

* JRC-IDEES dataset versioning implementation

* Remove obsolete script for retrieving jrc idees data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Scigrid gas network - versioning implementation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* doc: scigrid dataset updates to versions.csv

* Synthetic electricity demand versioning

* Copernicus Global land service versioning

* changing not-supported to might-work flags for some datasets

* Ship raster - versioning implementation

* EEZ dataset versioning implementation

* NUTS3 population dataset versioning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GDP per capita dataset versioning

* WorldPop population count dataset versioning

* EEA ghg emissions data versioning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GEBCO versioning

* Attributed Ports data versioning

* Updating URLs

* CORINE dataset versioning

* Emobility dataset versioning

* Common function to unzip folders

* H2 salt caverns data versioning

* Update primary link for GHG emissions

* Primary link update for population count dataset

* Lau regions data versioning

* Data versioning for Aquifer data BGR

* Updating archive link for lau regions

* Retrieving corine dataset from primary datasource

* OSM boundaries primary data versioning

* OSM boundaries archive link update

* rename function

* code: Fix enspreso_biomass dataset name

* code: Improve WDPA dataset handling

* code: Simplify zip file handling

* code: Simplify eurostat balances access

* code: Centrally configure http storage provider

* code: Fix typo

* code: Enable config of prepare_kwargs

* Bug fix - remove duplicate input entries for rule build_shapes

* Removing check for retrieve flag

* Moving corine Apikey from file to config parameter

* Polishing the code

* Code polishing for aquifer data set

* code: Remove unnecessary code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* code: simplify worldbank retrieval rule

* change indexing method for rules

* Remove code block for retrieving databundle

* modifying the way to unzip folders

* Code modifications

* doc updates

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Script license

* License correction

* Doc formatting

* code: Cleanup GEBCO retrieval

* code: Always copy storage(...) retrieved data, manage keep_local centrally

* code: remove params from retrieve_jrc_idees

* code: remove params from retrieve_gas_infrastructure

* code: remove params for retrieve_eez

* code: Remove params for retrieve_osm_boundaries

* code: Deprecate enable::retrieve, enable::retrieve_databundle, enable::retrieve_cost_data

* code: addendum to centrally managing `keep_local` setting for storage(...)

* code: formatting

* doc: Update configtables for data:: config options

* code: Cleanup old comments

* doc: remove duplicate dataset

* doc: Clarify `url` entry in data/versions.csv

* Code cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Adding comments

* Versioning for dh_areas

* Data versioning for geothermal heat utilisation potentials

* Update links for archive versions of dh_areas and geothermal heat utilisation potentials

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* doc: Configuration entries for dh_areas and geothermal_heat_utilisation_potentials

* code: Remove obsolete databundle retrieval

* doc: Update retrieve.rst

* doc: Remove outdated information and sites

* doc: Update PR template with hint to instruction on data versioning

* code: Fix config["enable"]["retrieve"] artifacts

* code: Update paths for accessing retrieved gas network

* code: Remove obsolete script

* code: Remove validate_checksum as it keeps failing from too many requests

* code: Fix links for web archive versions

* doc: Update formatting

* doc: try different table formatting

* code: Increase retries on retrieve

* code: Fix in biomass potentials

* code: Modify code to reference variable instead of index

* code: Bug fix

* code: correction to PyPSA set_option

* remove conda in snakemake rule

* code: Set max_requests_per_second

* code: Adding versioning for JRC Ardeco dataset

* code: JRC Ardeco URL updates

* code: plot balance map bug fix

* Update pypsa version to 0.33.2

* code: Cleanup ARDECO retrieval

* code: Fix ARDECO retrieval rules

* code: Adjust http storage limits to avoid Zenodo rate limit

* debug: Extract information about Zenodo cert

* Switch to primary sources for BFS data to avoid web archive retry limits

* code: See if we can extract the Zenodo rate limit

* code: Only request head

* code: Align retrieve_corine with rest of code

* code: Remove superflous parts

* code: Add modified snakemake storage plugin

* code: Fix typo

* wip

* WIP stuff

* code: Bugfix workflow

* nuclear approach to rate limiting errors

* code: Fix EEZ retrieval from archive

* feat: Add snakemake storage plugin for downloading zenodo files

with ratelimit backoff, concurrent downloading and progress bars.

* Update retrieve.smk to rely on zenodo storage plugin

* Make pre-commit happy

* fix url substring sanitization

* Use tqdm-loggable to avoid flooding github logs

* Add retries option to zenodo storage provider

* fix: merge conflict to accomodate new rule build_dh_areas

* fix lau_regions zenodo url

* Re-add zenodo checksums (retrieved from api) and pull retrying into storage plugin

* Replace aiohttp with httpx

better connection pooling and timeout handling.

* Release the stream a tiny bit earlier

* fix: do cache data_versions

for lru_cache to work the method needs to be at the toplevel.

* Allow deactivating the cache and skipping all remote checks

Use these settings for ci, since we already cache the resulting data folder.

* Derive from settings from SettingsBase and clean up some docs

* Revert "debug: Extract information about Zenodo cert"

This reverts commit c1edb28.

* fix: address unfounded fears

* improve docstrings

* feat: improve checksumming by allowing any digest

* Update README.md

* Expand user in `cache` directory

* fix: missed fixes

* cleanup

* refactor: move storage plugin to own repo

* refactor: rename `version_latest` to `version`

* feat: allow passing corine secret via envvar

* refactor: move zenodo cli to dedicated `utils` dir

* Skip storage plugin cache in CI

* feat: extract aquifer_data onaccess

saves space.

* fix: use vsizip to transparently decompress nuts3 2021

* fix: save synthetic electricity demand as csv.gz

* fix: clean conda caches after update

* Revert "fix: save synthetic electricity demand as csv.gz"

This reverts commit 0ebc02c.

* Revert "fix: use vsizip to transparently decompress nuts3 2021"

This reverts commit 9e6cd7f.

* Revert "feat: extract aquifer_data onaccess"

This reverts commit 6de88a2.

* remove tenacity and old checksum/ internet helpers

* Adjust file paths to work on Windows

Minor changes:
- set paths to always use posix format

* Fix file paths on windows nicely (#71)

* fix: use string paths with / even on windows

* feat: Add pixi toml and lock

* fix: fix retrieve_worldbank_urban_population

* Update rules/build_sector.smk

* Revert "feat: Add pixi toml and lock"

This reverts commit 9c9bf97.

* code: Remove quotation marks in config

* Update comment

* Update storage plugin config

* code: Update pixi dependencies

* code: Fix wildcard

* Update rules/build_electricity.smk

Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* Update doc/release_notes.rst

Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* update `pixi.lock`

* remove snakemake-storage-plugin-cached-http install from pypi

* Revert "update `pixi.lock`"

This reverts commit 6cecd76.

* revert updated lock file

* fix pre-commit

---------

Co-authored-by: Sermisha <sermisha.narayana@openenergytransition.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sermisha <156903227+SermishaNarayana@users.noreply.github.com>
Co-authored-by: Lukas Trippe <lkstrp@pm.me>
Co-authored-by: Jonas Hoersch <jonas.hoersch@openenergytransition.org>
Co-authored-by: Jonas Hörsch <coroa@posteo.de>
Co-authored-by: Eric Nitschke <eric.nitschke@tuhh.de>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>

Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>